Conventional disk based persistent storage requires expensive storage systems such as Storage Area Network/Network attached Storage (SAN/NAS) hardware to reliably store data without single point of failure. However, the SAN/NAS hardware cannot provide performance and throughput that can satisfactorily address the growing demands of the enterprise/cloud applications.
A typical enterprise application server environment supports the deployment of software applications to a cluster of processing nodes (variously referred to herein as machine nodes). To provide high-availability in the event of abnormal application or node behavior, a particular application may require its data to be persisted in some manner, which in turn typically requires the use of a backend filesystem or similar persistent storage medium that is accessible to all of the nodes.
However, the above approach generally does not scale well with increasing number of processing nodes. For example, a set of nodes may attempt to persist data to the backend filesystem at a rate that collectively exceeds the maximum rate at which the filesystem can actually write the data. In such situations, adding yet more nodes to the cluster provides little benefit, since the applications must wait for the backend filesystem to complete its write operations. These are the generally areas that embodiments of the invention are intended to address.